SEQUENCE LABELING TASKS

Filtered Semi-Markov CRF (2023.findings-emnlp)

Challenge:	Existing early-exit mechanisms are designed for sequence-level tasks, rather than sequence labeling.
Approach:	They propose to extend sentence-level early-exit to accelerate inference of PTMs . they propose a token-level mechanism that allows partial tokens to exit early at different layers .
Outcome:	The proposed approach can save up to 66%75% inference cost with minimal performance degradation.

Copied to clipboard

Urchade Zaratiana, Nadi Tomeh, Niama El Khbir, Pierre Holat, Thierry Charnois

Information Extraction & Text Mining

Challenge:	Existing methods for sequence labeling tasks such as Named Entity Recognition (NER) suffer from quadratic complexity over sequence length and poor performance compared to CRF.
Approach:	They propose a variant of Semi-Markov CRF that incorporates a filtering step to eliminate irrelevant segments, reducing complexity and search space.
Outcome:	The proposed method outperforms both CRF and Semi-CRF on several NER benchmarks while being significantly faster.

An Effective Incorporating Heterogeneous Knowledge Curriculum Learning for Sequence Labeling (2025.acl-short)

Copied to clipboard

Xuemei Tang, Jun Wang, Qi Su, Chu-Ren Huang, Jinghang Gu

Overcoming Catastrophic Forgetting in Massively Multilingual Continual Learning (2023.findings-acl)

Challenge:	Existing approaches to enhance sequence labeling models require data heterogeneity and additional modules.
Approach:	They propose a dual-stage curriculum learning framework specifically designed for sequence labeling tasks.
Outcome:	The proposed model improves training and accelerates training, mitigating the slow training issue of complex models.

Copied to clipboard

Genta Winata, Lingjue Xie, Karthik Radhakrishnan, Shijie Wu, Xisen Jin, Pengxiang Cheng, Mayank Kulkarni, Daniel Preotiuc-Pietro

Multilinguality

Challenge:	Existing methods to handle catastrophic forgetting fail to retain knowledge learnt in the past when sudden shifts occur in training data distributions.
Approach:	They propose a learning rate scheduling method that preserves new information without strongly overwriting past knowledge.
Outcome:	The proposed method preserves new information without overwriting past knowledge in a multilingual continuous learning framework.

Regression-Free Model Updates for Spoken Language Understanding (2023.acl-industry)

Copied to clipboard

Andrea Caciolai, Verena Weber, Tobias Falke, Alessandro Pedrani, Davide Bernardi

Information Retrieval

Challenge:	Recent work has proposed methods for minimizing regressions caused by model updates . focus is on spoken language understanding models, which are unexplored .
Approach:	They propose a focal distillation technique to reduce regressions in goal-oriented dialog systems . they also evaluate its effectiveness for key language understanding tasks .
Outcome:	The proposed technique outperforms naive supervised training in mislabeled data and label expansion settings.

Improving BERT with Syntax-aware Local Attention (2021.findings-acl)

Copied to clipboard

Zhongli Li, Qingyu Zhou, Chao Li, Ke Xu, Yunbo Cao

Parallel Attention Network with Sequence Matching for Video Grounding (2021.findings-acl)

Challenge:	Recent studies show that attention-based models benefit from more focused attention over local regions.
Approach:	They propose a syntax-aware local attention which restrains attention over syntactically relevant words.
Outcome:	The proposed model performs better on all benchmark datasets, including sentence classification and sequence labeling tasks.

Copied to clipboard

Hao Zhang, Aixin Sun, Wei Jing, Liangli Zhen, Joey Tianyi Zhou, Siow Mong Rick Goh

Multimodality

Challenge:	Existing approaches to video grounding are sensitive to quality of proposals and inefficient because all proposal-query pairs are compared.
Approach:	They propose a Parallel Attention Network with Sequence matching to capture selfmodal contexts and cross-modal attentive information between video and text.
Outcome:	The proposed approach is superior to state-of-the-art methods on three datasets.

”Diversity and Uncertainty in Moderation” are the Key to Data Selection for Multilingual Few-shot Transfer (2022.findings-naacl)

Copied to clipboard

Shanu Kumar, Sandipan Dandapat, Monojit Choudhury

Pooled Contextualized Embeddings for Named Entity Recognition (N19-1)

Challenge:	Existing approaches for few-shot transfer show significant gain over zero-shot transfers . language resource distribution is skewed across the world's languages . proposed methods use multiple measures such as data entropy and gradient embedding .
Approach:	They propose a loss embedding method for sequence labeling tasks that induces diversity and uncertainty sampling similar to gradient embeddment.
Outcome:	The proposed methods outperform baseline methods for POS tagging, NER, and NLI tasks for up to 20 languages.

Copied to clipboard

Alan Akbik, Tanja Bergmann, Roland Vollgraf

Robust Multilingual Part-of-Speech Tagging via Adversarial Training (N18-1)

Challenge:	Contextual string embeddings are a recent type of word embeddable that are useful for sequence labeling tasks.
Approach:	They propose a method that dynamically aggregates contextualized embeddings of each unique string . they then use a pooling operation to distill a ”global” word representation from all contextualized instances .
Outcome:	The proposed method improves state-of-the-art for named entity recognition tasks.

Copied to clipboard

Michihiro Yasunaga, Jungo Kasai, Dragomir Radev

Syntactic Text Processing

Challenge:	Adversarial training (AT) is a powerful regularization method for neural networks, aiming to achieve robustness to input perturbations.
Approach:	They propose and analyze a neural POS tagging model that exploits adversarial training by training on unmodified and adversarials.
Outcome:	The proposed model improves overall tagging accuracy and prevents over-fitting in low resource languages and boosts tabbing accuracy for rare / unseen words.

Prompt Perturbation Consistency Learning for Robust Language Models (2024.findings-eacl)

Copied to clipboard

Yao Qiang, Subhrangshu Nandi, Ninareh Mehrabi, Greg Ver Steeg, Anoop Kumar, Anna Rumshisky, Aram Galstyan

Contextual String Embeddings for Sequence Labeling (C18-1)

Challenge:	Large language models have demonstrated impressive performance on a number of natural language processing tasks, such as question answering and text summarization.
Approach:	They propose a method to reduce the performance drop of large language models by regularizing the divergence between losses from clean and perturbed samples.
Outcome:	The proposed approach recovers on average 59% and 69% of the performance drop for IC and SF tasks while using ten times fewer augmented data samples.

Copied to clipboard

Alan Akbik, Duncan Blythe, Roland Vollgraf

Contributions of Transformer Attention Heads in Multi- and Cross-lingual Tasks (2021.acl-long)

Challenge:	Recent advances in language modeling have made it viable to model language as distributions over characters.
Approach:	They propose to leverage internal states of a trained character language model to produce a new type of word embeddings.
Outcome:	The proposed embeddings outperform the state-of-the-art on four classic sequence labeling tasks.

Copied to clipboard

Weicheng Ma, Kai Zhang, Renze Lou, Lili Wang, Soroush Vosoughi

ToPro: Token-Level Prompt Decomposition for Cross-Lingual Sequence Labeling Tasks (2024.eacl-long)

Challenge:	Prior research has found that only a few attention heads are important in each mono-lingual NLP task and pruning the remaining heads leads to comparable or improved performance of the model.
Approach:	They examine the relative importance of attention heads in Transformer-based models to aid their interpretability in cross-lingual and multi-lingual tasks.
Outcome:	The proposed model performs better with the remaining heads pruned than with the other models, the authors show .

Copied to clipboard

Bolei Ma, Ercong Nie, Shuzhou Yuan, Helmut Schmid, Michael Färber, Frauke Kreuter, Hinrich Schuetze

Uncertainty-Aware Label Refinement for Sequence Labeling (2020.emnlp-main)

Challenge:	Prompt-based methods have been successfully applied to multilingual pretrained language models for zero-shot cross-lingual understanding.
Approach:	They propose a prompt-based method for token-level sequence labeling tasks . they propose to decompose an input sentence into single tokens and apply one prompt template to each token.
Outcome:	The proposed method outperforms Vanilla fine-tuning and Prompt-Tuning in zero-shot cross-lingual transfer . the method also attains state-of-the-art performance when employed with the mT5 model .

Copied to clipboard

Tao Gui, Jiacheng Ye, Qi Zhang, Zhengyan Li, Zichu Fei, Yeyun Gong, Xuanjing Huang

Green & Sustainable NLP

Challenge:	Conditional random fields (CRF) for label decoding have been a problem for many tasks.
Approach:	They propose a two-stage label decoding framework that model long-term label dependencies while being much more computationally efficient.
Outcome:	The proposed method outperforms the CRF-based methods and greatly accelerates the inference process.

DyLex: Incorporating Dynamic Lexicons into BERT for Sequence Labeling (2021.emnlp-main)

Copied to clipboard

Baojun Wang, Zhao Zhang, Kun Xu, Guang-Yuan Hao, Yuyang Zhang, Lifeng Shang, Linlin Li, Xiao Chen, Xin Jiang, Qun Liu

GCDT: A Global Context Enhanced Deep Transition Architecture for Sequence Labeling (P19-1)

Challenge:	Existing approaches to integrate lexical knowledge into deep learning models are limited by large-scale dynamic lexicons.
Approach:	They propose a plug-in lexicon incorporation approach for BERT based sequence labeling tasks . they adopt word-agnostic tag embeddings to avoid re-training the representation .
Outcome:	The proposed framework achieves new SOTA even with large scale lexicons, the authors show . they adopt word-agnostic tag embeddings to avoid re-training the representation .

Copied to clipboard

Yijin Liu, Fandong Meng, Jinchao Zhang, Jinan Xu, Yufeng Chen, Jie Zhou

Chunking

Challenge:	Existing systems for sequence labeling are limited by shallow connections between consecutive hidden states and insufficient modeling of global information.
Approach:	They propose a global context enhanced deep transition architecture for sequence labeling . they deepen the state transition path at each position in a sentence and assign tokens with global representations .
Outcome:	The proposed architecture outperforms the best reported results on two standard sequence labeling tasks.

ConVEx: Data-Efficient and Few-Shot Slot Labeling (2021.naacl-main)

Copied to clipboard

Matthew Henderson, Ivan Vulić

Single Model Ensemble using Pseudo-Tags and Distinct Vectors (2020.acl-main)

Challenge:	ConVEx is an efficient pretraining and fine-tuning neural approach for slot-labeling dialog tasks.
Approach:	They propose an efficient pretraining and fine-tuning neural approach for slot-labeling dialog tasks that uses a pairwise cloze task and reddit data.
Outcome:	The proposed approach is well aligned with its intended use on slot-labeling tasks and can be used across a range of domains and data sets.

Copied to clipboard

Ryosuke Kuwabara, Jun Suzuki, Hideki Nakayama

Challenge:	Existing methods for model ensembles require time, memory, and management effort to perform tasks.
Approach:	They propose a method that replicates the effects of a model ensemble with a single model.
Outcome:	The proposed method emulates or outperforms a traditional model ensemble with 1/K-times fewer parameters on text classification and sequence labeling tasks.

Switch Point biased Self-Training: Re-purposing Pretrained Models for Code-Switching (2021.findings-emnlp)

Copied to clipboard

Parul Chopra, Sai Krishna Rallabandi, Alan W Black, Khyathi Raghavi Chandu

Multilinguality

Challenge:	Code-switching (CS) is a phenomenon of switching between multiple languages . current models cannot handle CS due to lack of annotated data and limited resources.
Approach:	They propose a self-training method to repurpose existing models using a switch-point bias by leveraging unannotated data to reduce the gap between the switch point performance and retain overall performance on two distinct language pairs.
Outcome:	The proposed model reduces the gap between the switch point performance while retaining the overall performance on two distinct language pairs.

Unsupervised Data Augmentation with Naive Augmentation and without Unlabeled Data (2021.emnlp-main)

Copied to clipboard

David Lowell, Brian Howard, Zachary C. Lipton, Byron Wallace

Low-Resource NLP

Challenge:	Unsupervised Data Augmentation (UDA) is a semisupervised learning method that penalizes differences between a model's predictions on unlabeled examples and corresponding 'noised' examples produced via data augmentation.
Approach:	They propose to use a consistency loss to penalize differences between models' predictions on unlabeled and unlabed examples to enforce consistency between models and their perturbed counterparts.
Outcome:	The proposed method is able to penalize differences between models' outputs on unlabeled and unlabed examples without complex data augmentation.

Fine-grained Knowledge Fusion for Sequence Labeling Domain Adaptation (D19-1)

Copied to clipboard

Huiyun Yang, Shujian Huang, Xin-Yu Dai, Jiajun Chen

An Exploration of Arbitrary-Order Sequence Labeling via Energy-Based Inference Networks (2020.emnlp-main)

Challenge:	Existing domain adaptation methods focus on the adaptation from the source domain to the entire target domain without considering the diversity of individual sample samples.
Approach:	They propose a fine-grained knowledge fusion model with the domain relevance modeling scheme to control the balance between learning from the target domain data and learning from a source domain model.
Outcome:	The proposed model outperforms baselines and state-of-the-art models on three sequence labeling tasks.

Copied to clipboard

Lifu Tu, Tianyu Liu, Kevin Gimpel

Challenge:	Recent work shows that conditional random fields (CRFs) perform well in sequence labeling tasks.
Approach:	They propose several high-order energy terms to capture dependencies among labels in sequence labeling . they use convolutional, recurrent, and self-attention networks to construct these energy terms .
Outcome:	The proposed approach improves on four sequence labeling tasks while having the same decoding speed as simple classifiers.

bgGLUE: A Bulgarian General Language Understanding Evaluation Benchmark (2023.acl-long)

Copied to clipboard

Momchil Hardalov, Pepa Atanasova, Todor Mihaylov, Galia Angelova, Kiril Simov, Petya Osenova, Veselin Stoyanov, Ivan Koychev, Preslav Nakov, Dragomir Radev

Phrase Grounding by Soft-Label Chain Conditional Random Field (D19-1)

Challenge:	bgGLUE is a benchmark for evaluating language models on natural language understanding (NLU) tasks in Bulgarian.
Approach:	They propose to use a benchmark to evaluate language models on NLU tasks in Bulgarian.
Outcome:	The proposed model performs well on sequence labeling tasks, but there is room for improvement for tasks that require more complex reasoning.

Copied to clipboard

Jiacheng Liu, Julia Hockenmaier

Multimodality

Challenge:	Existing methods to ground entities depend on inference or non-differentiable losses.
Approach:	They propose a phrase grounding task that grounds entities to corresponding regions in an image . they use neural chain Conditional Random Fields to model dependencies among regions .
Outcome:	The proposed method is based on a dataset of the Flickr30k Entities dataset.

Evaluating Performance of Pre-trained Word Embeddings on Assamese, a Low-resource Language (2024.lrec-main)

Copied to clipboard

Dhrubajyoti Pathak, Sukumar Nandi, Priyankoo Sarmah

When Generative Adversarial Networks Meet Sequence Labeling Challenges (2024.emnlp-main)

Challenge:	Word embeddings are not explored in high-resource languages such as Assamese, where resources are limited.
Approach:	They propose to use assamese pre-trained word embeddings for sequence labeling tasks such as Parts-of-speech and Named Entity Recognition to evaluate their performance.
Outcome:	The proposed embeddings outperform the existing methods on Parts-of-speech and Named Entity Recognition tasks.

Copied to clipboard

Yu Tong, Ge Chen, Guokai Zheng, Rui Li, Jiang Dazhi

Robustness in NLP

Challenge:	Existing approaches for sequence labeling use a feature extractor and sequence tagger . a recent study shows that SLGAN is versatile and highly effective .
Approach:	They propose a framework that harnesses the capabilities of Generative Adversarial Networks to address sequence labeling challenges.
Outcome:	The proposed framework exhibits strong adaptability to various sequence labeling tasks.

Recipes for Sequential Pre-training of Multilingual Encoder and Seq2Seq Models (2023.findings-acl)

Copied to clipboard

Saleh Soltan, Andy Rosenbaum, Tobias Falke, Qin Lu, Anna Rumshisky, Wael Hamza

A Fine-grained Chinese Software Privacy Policy Dataset for Sequence Labeling and Regulation Compliant Identification (2022.emnlp-main)

Challenge:	Pre-trained encoder-only and sequence-to-sequence models are computationally expensive.
Approach:	They propose a recipe to initialize one model from the other to improve pre-training efficiency.
Outcome:	The proposed method matches the performance of a from-scratch model with a multilingual encoder while reducing the total compute cost by 27%.

Copied to clipboard

Kaifa Zhao, Le Yu, Shiyao Zhou, Jing Li, Xiapu Luo, Yat Fei Aemon Chiu, Yutong Liu

DemoNSF: A Multi-task Demonstration-based Generative Framework for Noisy Slot Filling Task (2023.findings-emnlp)

Challenge:	Existing datasets that ignore law requirements are limited to English.
Approach:	They construct a Chinese privacy policy dataset that can be used to analyze software privacy policies.
Outcome:	The proposed dataset includes 483 Chinese Android privacy policies, over 11K sentences, and 52K fine-grained annotations.

Copied to clipboard

Guanting Dong, Tingfeng Hui, Zhuoma GongQue, Jinxu Zhao, Daichi Guo, Gang Zhao, Keqing He, Weiran Xu

SeqVAT: Virtual Adversarial Training for Semi-Supervised Sequence Labeling (2020.acl-main)

Challenge:	Recent prompt-based generative frameworks have shown impressive capabilities in sequence labeling tasks.
Approach:	They propose a multi-task demonstration-based generative framework for noisy slot filling that captures input perturbations at different granularities.
Outcome:	The proposed framework outperforms baseline methods and achieves strong generalization.

Copied to clipboard

Luoxin Chen, Weitong Ruan, Xinyue Liu, Jianhua Lu